test: add code agent eval framework with 4 functional test cases#216
Closed
mysticgohan1 wants to merge 1 commit into
Closed
test: add code agent eval framework with 4 functional test cases#216mysticgohan1 wants to merge 1 commit into
mysticgohan1 wants to merge 1 commit into
Conversation
Add eval infrastructure for the code agent pipeline (issue → fix → PR) and four test cases exercising different patterns: - 001-sort-direction: trivial Python bug fix (baseline) - 002-docs-guidance: docs-only AGENTS.md edit (Go repo) - 003-extract-shared: multi-file refactor with planted bug (Python) - 004-shell-validation: shell script env var validation gap Infrastructure: eval.yaml with 5 judges (pr_created, code_quality, forbidden_labels, max_turns, max_cost), capture-code-output.sh hook, label support in setup-fixture.sh, env var export in run-fullsend.sh. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: guy oron <goron@redhat.com>
|
Thank you for your interest in contributing, @mysticgohan1. This project uses a vouch system for first-time contributors. Before submitting a pull request, you need to be vouched by a maintainer. To get vouched:
See CONTRIBUTING.md for details. |
Functional tests did not runFunctional tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the |
PR Summary by QodoAdd code-agent eval harness and 4 functional edge-case fixtures
AI Description
Diagram
High-Level Assessment
Files changed (39)
|
Code Review by Qodo
Context used✅ Compliance rules (platform):
55 rules 1. DEPLOY_TARGET not validated
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
| @@ -0,0 +1,9 @@ | |||
| #!/usr/bin/env bash | |||
| # Deploy script — pushes build artifacts to the target server. | |||
| set -eo pipefail | |||
There was a problem hiding this comment.
1. deploy_target not validated 📜 Skill insight ⛨ Security
deploy.sh proceeds to run rsync even when DEPLOY_TARGET is unset/empty, which is a fail-open validation gate and can cause unintended behavior. The script should explicitly reject missing configuration before performing deployment actions.
Agent Prompt
## Issue description
`eval/code/cases/004-shell-validation/repo/scripts/deploy.sh` uses `${DEPLOY_TARGET:-}` and does not validate that `DEPLOY_TARGET` is set/non-empty before running deployment logic, so the script can proceed in a permissive (fail-open) way when configuration is absent.
## Issue Context
Other scripts in the same case (e.g., `setup.sh` / `rollback.sh`) already follow a strict env-var validation pattern.
## Fix Focus Areas
- eval/code/cases/004-shell-validation/repo/scripts/deploy.sh[1-9]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| @@ -0,0 +1,188 @@ | |||
| name: code-eval | |||
There was a problem hiding this comment.
3. No linked issue reference 📜 Skill insight § Compliance
This PR introduces a substantial new eval framework and multiple test case fixtures but does not link to an authorizing issue in the PR description. For non-trivial changes, explicit issue authorization is required.
Agent Prompt
## Issue description
Non-trivial PRs must link to an authorizing issue (e.g., "Fixes #123" / "Refs #123"), but the PR description contains no linked issue reference.
## Issue Context
The diff adds a full new eval (`eval/code/eval.yaml`) and multiple new case repositories, indicating a non-trivial scope.
## Fix Focus Areas
- eval/code/eval.yaml[1-10]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| mkdir -p "$OUTPUT_DIR" | ||
| PR_STATE_FILE="${OUTPUT_DIR}/pr-state.json" | ||
|
|
||
| # Find PRs on the ephemeral repo. The post-script creates a PR whose |
There was a problem hiding this comment.
4. Wrong pr captured 🐞 Bug ≡ Correctness
eval/scripts/capture-code-output.sh selects the first PR returned by gh pr list without verifying it is the PR created for the fixture issue, so when multiple PRs exist it may capture an unrelated diff and mis-score the case.
Agent Prompt
### Issue description
`capture-code-output.sh` claims it finds the PR whose body contains `Closes #<FIXTURE_NUMBER>`, but it actually captures `.[0]` from `gh pr list` with no association check. If multiple PRs exist (retries, multiple agent PRs, etc.), judges may evaluate the wrong PR diff.
### Issue Context
The post-script always includes `Closes #${ISSUE_NUMBER}` in the PR body, so the capture hook can reliably filter on that marker.
### Fix Focus Areas
- eval/scripts/capture-code-output.sh[24-34]
### Implementation notes
- Fetch the PR list and select the first PR whose `.body` contains `Closes #${FIXTURE_NUMBER}` (case-insensitive is fine).
- If no match is found, write `{error: "No PR found matching fixture"}` (optionally include fixture number and list size) and exit 0 like today.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| [[ -n "${GOOGLE_APPLICATION_CREDENTIALS:-}" ]] && echo "GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}" | ||
| } > "$ENV_FILE" | ||
|
|
||
| export REPO_FULL_NAME="${EPHEMERAL_REPO}" |
There was a problem hiding this comment.
5. Incorrect issue_number exported 🐞 Bug ☼ Reliability
eval/scripts/run-fullsend.sh exports ISSUE_NUMBER=${FIXTURE_NUMBER} for every fixture type, so PR
fixtures will incorrectly receive ISSUE_NUMBER set to the PR number, which conflicts with code-skill
scripts that validate ISSUE_NUMBER against GITHUB_ISSUE_URL.
Agent Prompt
### Issue description
`run-fullsend.sh` now unconditionally exports `ISSUE_NUMBER` (and `REPO_FULL_NAME`) after the fixture-type switch. For non-issue fixtures this can introduce semantically wrong inputs (e.g., PR number in ISSUE_NUMBER).
### Issue Context
The code skill’s pre-script enforces that `ISSUE_NUMBER` is a positive integer and matches `GITHUB_ISSUE_URL`. Exporting `ISSUE_NUMBER` in PR contexts is a compatibility hazard and can confuse downstream logic that keys off `ISSUE_NUMBER`.
### Fix Focus Areas
- eval/scripts/run-fullsend.sh[54-65]
- eval/scripts/run-fullsend.sh[73-75]
### Implementation notes
- Move the exports into the `case "$FIXTURE_TYPE"` block:
- `issue) export REPO_FULL_NAME=...; export ISSUE_NUMBER=...;;`
- `pull_request) export REPO_FULL_NAME=...; export PR_NUMBER=...; unset ISSUE_NUMBER;;`
- Keep writing the env-file as you do today; just make exported vars consistent with fixture type.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| FIXTURE_NUMBER="${FIXTURE_URL##*/}" | ||
| echo "Created issue: $FIXTURE_URL" | ||
|
|
||
| # Apply fixture labels if declared in input.yaml. |
There was a problem hiding this comment.
6. Label setup failures hidden 🐞 Bug ☼ Reliability
eval/scripts/setup-fixture.sh suppresses errors when creating and applying fixture labels, so a fixture can silently miss its intended labels and change agent behavior / label-based evaluation expectations without failing setup.
Agent Prompt
### Issue description
Fixture label creation and label application both redirect stderr and `|| true`, so label failures are silent and setup continues. That can invalidate a case (labels in input.yaml aren’t actually present) while still producing outputs.
### Issue Context
Cases now declare `fixture.labels` in input.yaml, so labels are part of the test fixture contract.
### Fix Focus Areas
- eval/scripts/setup-fixture.sh[108-118]
### Implementation notes
- Remove `2>/dev/null || true` for label creation/application, or at minimum:
- emit a warning to stderr and exit non-zero if applying labels fails.
- optionally verify labels were applied by reading back `gh issue view --json labels` and checking expected names.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| lint: | ||
| gofmt -l . | ||
|
|
||
| check-docs: |
There was a problem hiding this comment.
7. Docs check incomplete 🐞 Bug ≡ Correctness
In case 002, make check-docs only validates the Error Handling heading plus exception/traceback keywords, so it can pass even if the required logging-before-reraise or structured API error guidance is missing.
Agent Prompt
### Issue description
The case requires 4 specific topics in the new AGENTS.md section, but the repo’s `check-docs` target only checks for 2 of them (plus the heading). This weakens the fixture’s self-verification.
### Issue Context
Both the issue text and annotations list the required topics; `check-docs` should reflect that contract.
### Fix Focus Areas
- eval/code/cases/002-docs-guidance/repo/Makefile[9-14]
- eval/code/cases/002-docs-guidance/input.yaml[12-20]
- eval/code/cases/002-docs-guidance/annotations.yaml[15-20]
### Implementation notes
- Add grep checks for:
- logging before re-raising (e.g., `logger.error` / `log` / `re-raise` wording)
- structured API errors (e.g., `structured` + `error response`, `JSON`, etc.)
- (Optional) also assert placement: Error Handling appears after `## Code Style` and before `## Testing`.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the code agent eval framework and four functional test cases covering distinct patterns observed in the 30-issue benchmark:
sorted()missingreverse=Trueformat_error()witherr/exctypodeploy.shmissing env var validationInfrastructure
eval/code/eval.yaml: 5 judges (pr_created, code_quality LLM judge, forbidden_labels, max_turns, max_cost)eval/scripts/capture-code-output.sh: after-each hook to capture PR stateeval/scripts/setup-fixture.sh: added optionalfixture.labelssupport (backward-compatible)eval/scripts/run-fullsend.sh: exportsREPO_FULL_NAME+ISSUE_NUMBERfor code agent (backward-compatible)Verification
eval/lint-cases.sh codepasseseval/lint-cases.sh triageandeval/lint-cases.sh reviewstill pass (no regressions)Test plan
eval/lint-cases.sh codepasseseval/lint-cases.sh triage,eval/lint-cases.sh review)pytestshows 1 failure (test_highest_priority_first), fix isreverse=Truemake check-docsfails (missing section), fix is adding## Error Handlingpytestshows 3 failures (webhookNameError), fix is extracting sharederrors.pytest_missing_vars.shshows 1 failure (deploy.sh), fix is adding validation loop🤖 Generated with Claude Code